resource delay in db : Async wait for [Cancelled]#233
Closed
resource delay in db : Async wait for [Cancelled]#233
Conversation
Codecov ReportAttention: Patch coverage is 📢 Thoughts on this report? Let us know! |
nishika26
changed the title
nishika26
changed the title
avirajsingh7
approved these changes
Jun 18, 2025
| request, | ||
| payload, | ||
| ) | ||
| # Start the background task asynchronously |
Collaborator
There was a problem hiding this comment.
Any reason it is here?
Collaborator
Author
There was a problem hiding this comment.
Leaving it for any future reference, although it is self explanatory but still just in case
AkhileshNegi
requested changes
Jun 18, 2025
nishika26
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Target issue is #217
Explain the motivation for making this change. What existing problem does the pull request solve?
we'll be implementing a temporary solution using asyncio.wait_for(). This will allow us to:
Start the background task (like creating the collection or updating the vector store).
Wait for the task to complete within a specified timeout. If the task finishes within the timeout, we send a success response. This ensures that the response, which includes the collection ID, will only be sent after the task is completed, so the collection is already present in the database and we avoid 'no row found' errors.
After using Python's time.time() to measure the time taken by the background task, it took around 7.67 seconds to complete when I uploaded an example file used in Dalgo's data analysis feature. Given this, we can estimate a timeout of approximately 15-20 seconds. Any task that finishes within that window will trigger a response before the timeout.
Notes
A few additional changes I’ve made:
Updated def create_collection to async def create_collection since await can only be used within async def functions. Reference
Switched to using an async test client to properly test asynchronous route handlers.
Installed the pytest-asyncio library, which is required for running async tests with pytest.
Logging few important parameters such as -
INFO - Collection created: 425111c7-25a3-42d3-b373-c97202d9aad5 | Time: 7.514757871627808s | Files: 1 |Sizes:[137.57] KB |Types: ['txt']To log file size(s) of files being uploaded, this logic was used to first calculate the file size and then the logic was added to
core/cloud/storage.py